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ABSTRACT 

This  paper  describes  a  novel  Mixed  Reality  [1]  technique  for  robust ,  real-time  chromakey  processing  for 
training  applications  using  software  and  off-the-shelf  video  hardware.  This  technique  has  been  coined  and 
patented  by  the  author  as  'Fused  Reality.  "  Until  now  chromakeying  has  been  conducted  using  dedicated 
hardware,  imposing  substantial  restrictions  on  the  visual  environments  that  will  support  chromakey. 
Variations  on  the  traditional  chromakey  setup,  such  as  using  retroreflective  screens  and  light-emitting 
cameras,  can  overcome  some  of  the  technique ’s  original  drawbacks  (such  as  lighting  difficulties),  but  they  can 
introduce  new  problems  as  well  (i.e.,  a  user's  hand  can  obstructing  the  light  projected  from  the  head-mounted 
source  can  create  an  ill-defined  silhouette).  The  novel  chromakey  method  introduced  in  this  paper  is  applied 
to  training  helicopter  aircrew  personnel  using  a  prototype  simulator,  the  Prototype  Aircrew  Virtual 
Environment  Training  (PAVET)  System.  The  ultimate  goal  of  the  PAVET  will  be  to  provide  training  to  Navy 
aircrewmen  in  all  operational  aspects  of  the  MH-60S,  including  aerial  gunnery,  search  and  rescue,  and 
vertical  replenishment.  A  key  requirement  for  these  types  of  tasks  is  for  trainees  to  see  and  manipulate 
physical  objects  (e.g.,  a  jammed  gun)  at  close  range  while  viewing  an  interactive  flight  and  shipboard 
environment.  In  order  to  satisfy  space  and  cost  constraints,  some  physical  objects  that  the  aircrewmen 
physically  interact  with  (such  as  a  rescue  litter)  must  be  capable  of  being  sent  out  into  the  virtual  environment 
and  later  retrieved.  Fused  Reality  accomplishes  this  critical  feature  through  contraction  and  expansion  of 
specific  real-world  objects  as  they  move  away  and  toward  the  trainee.  Fused  Reality's  adaptive  color 
recognition  allows  for  realistic  set  lighting,  colors,  and  user  movement  and  positioning.  It  also  enables 
“lumakey"  -  preserving  only  pixels  that  are  above  a  brightness  threshold  and  rendering  all  others 
transparent,  allowing  for  extremely  compact  and  portable  training  devices.  Examples  of  Fused  Reality  such 
as  these  are  demonstrated  and  discussed. 


INTRODUCTION 

The  effectiveness  and  safety  of  complex,  multi-role  helicopter  platforms  such  as  the  Pave  Hawk  and  Seahawk 
require  that  the  cabin  crew  interact  seamlessly  with  the  flight  crew  and  a  dynamic  external  environment.  Due 
to  physical  constraints  and  fidelity  limitations,  however,  current  simulation  designs  fail  to  accommodate  a 
wide  range  of  training.  Fused  Reality  (coined  by  the  author)  is  a  Mixed  Reality  approach  that  employs  three 
proven  technologies  -  live  video  capture,  real-time  video  editing,  and  virtual  environment  simulation  - 
offering  a  quantum  jump  in  training  realism  and  capability.  Video  from  the  trainee’s  perspective  is  sent  to  a 
processor  that  preserves  near-space  (cabin  environment)  pixels  and  makes  transparent  the  far-space  (out-the- 
cabin)  pixels  using  blue  screen  imaging  techniques.  This  bitmap  is  overlaid  on  a  virtual  environment,  and  sent 
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to  the  trainee’s  helmet  mounted  display.  Thus  the  user  directly  views  the  physical  cabin  environment,  while 
the  simulated  outside  world  serves  as  a  backdrop. 

Fused  Reality  is  a  technique  conceived  at  STI  and  prototyped  in  response  to  the  training  requirements  called 
for  by  the  Navy’s  Prototype  Aircrew  Virtual  Environment  Trainer  (PA VET).  As  part  of  a  Navy  Phase  II 
SBIR,  STI  is  integrating  Fused  Reality  with  PA  VET  in  partnership  with  Binghamton  Simulator  Company, 
which  is  providing  the  physical  environment  (cabin,  gun,  etc.).  Fused  Reality  should  rapidly  find  application 
in  numerous  other  sectors,  such  as  the  medical,  commercial  aviation,  entertainment,  etc.  The  technique  is 
similar  in  certain  respects  to  the  blue-screening  technique  that  Hollywood  is  using  (employed  initially  by 
Hitchcock  in  Vertigo).  However,  Hollywood  processes  its  blue-screening  off-line  -  STI  is  conducting  it  in 
real-time  -  and  in  contrast  with  blue  screening  the  backdrop  required  by  Fused  Reality  allows  large  variations 
in  color  aberrations  and  lighting  intensity. 

Figure  1  is  a  basic  representation  of  Fused  Reality  showing  two  bitmap  layers  that  are  stacked  on  top  of  each 
other.  The  first  layer  is  video  imagery  (i.e.  the  user’s  immediate,  three-dimensional  environment)  sent  from 
the  user’s  micro-camera,  which  is  converted  to  a  bitmap.  The  second  layer  is  a  simulated,  three-dimensional 
(3D)  world  through  which  the  user  can  move  and  interact  with  -  a  head-tracker  communicates  user  head 
motion  to  this  virtual  world  so  that  the  fused  picture  of  both  worlds  is  consistent  with  user  motion.  The  two 
layers  can  be  processed  so  as  to  simulate  various  lighting  effects,  such  as  Night  Vision  Goggles,  or  low  light 
environments. 
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Figure  1:  Representation  of  Fused  Reality. 
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Helicopter  Cabin  Crew  Tasks  and  Training 

From  the  relatively  cramped  and  noisy  cabin  area  the  helicopter  crew  chief  may  have  to  perform  a  variety  of 
tasks,  including:  a)  Door  gunner  targeting,  b)  Cargo  hook  operations  monitoring,  including  emergency  load 
release,  c)  Hover  position  control  (fore/aft  and  lateral)  during  rescue  missions,  d)  Stokes  litter  deployment/ 
recovery,  e)  Helicopter  inflight  refueling  (HIFR)  procedures,  and  f)  Hoist  control  and  cargo  hook  release. 
Depending  on  the  mission  and  actual  platform,  the  crew  chief  may  be  called  on  to  execute  numerous  other 
tasks,  in  visual  environments  ranging  from  night  vision  goggles  (NVG’s)  to  brown-out  (blowing  sand  or  dirt). 

Due  to  the  difficulty  of  simulating  many  of  these  tasks  and  environments,  cabin  crew  training  usually 
transitions  directly  from  classroom  to  the  aircraft.  Aircraft  training  not  only  requires  the  presence  of  a  skilled, 
current  crew  chief  (which  strains  squadron  resources),  but  it  can  also  expose  novices  to  dangerous,  unfamiliar 
procedures  in  distracting  environments.  From  the  author’s  experience  (former  SH-60B  pilot),  the  difference 
between  performing  machine  hang-fire  procedures  in  the  classroom  and  in  the  aircraft  for  real  can  be  half  a 
crewman’s  foot. 

Candidate  Simulation  Approaches 

Several  candidate  approaches  are  discussed  here  that  were  considered  to  simulate  the  helicopter  cabin  crew 
environment.  With  a  Virtual  Reality  approach,  tactile  interaction  with  physical  objects  requires  distance 
sensing  between  the  human  touchpoints  (hand,  elbow,  etc.)  and  all  objects  that  can  be  touched.  This  becomes 
an  intractable  problem  for  complex  manual  tasks  such  as  machine  gun  disassembly.  Even  for  relatively  simple 
manual  tasks  such  as  flight  control  manipulation,  the  user  cannot  see  actual  positioning  of  limbs  and  hands 
(important  elements  for  manual  control)  unless  the  body  is  peppered  with  expensive  sensors.  Additionally, 
user  cannot  realistically  perceive  the  facial  expressions,  body  posture,  and  clothing  of  other  participants. 

With  the  CAVE  (Computer  Animated  Virtual  Environment)  approach,  a  hemispherical  dome  display  is  placed 
around  the  cabin  so  that  trainee  will  see  a  projected  environment  (with  the  naked  eye)  beyond  the  cabin 
airframe.  Disadvantages  of  technique  this  include:  1)  Large  volume  required;  2)  High-cost  displays  (a  display 
must  be  placed  wherever  an  image  is  to  appear);  3)  Real  objects  cannot  be  processed  (such  as  deploying  them 
into  the  virtual  environment,  discussed  later  in  this  paper),  and  4)  Large  power  requirements  (heat  generation, 
component  failure). 

CHROMAKEY  PROCESS 

Blue  screening  involving  human  filming  usually  employs  a  blue  or  green  backdrop,  since  skin  contains  little 
blue  or  green  hue.  These  backdrops  preclude  the  use  of  similar  colors  in  the  physical  environment.  Fused 
Reality  uses  magenta  as  the  target  color,  since  it  is  rarely  encountered  in  simulation  environments  (i.e.  the 
cockpit).  The  color  recognition  technique  used  in  Fused  Reality  can  accurately  distinguish  between  skin  tones 
and  magenta.  Figure  2  shows  an  image  of  magenta  material,  and  Figure  3  gives  pixel  scatterplots  of  the  red, 
green,  and  blue  (RGB)  components  comprising  the  color  in  Figure  2.  Due  to  non-uniformities  across  the 
material  surface,  as  well  as  sensor  artifacts  and  lens  artifacts  (note  the  darker  areas  in  the  corners  of  Figure  2), 
there  is  a  wide  variance  in  RGB  values.  In  order  to  algorithmically  define  the  target  color,  the  areas  of  the 
scatterplots  in  Figure  3  are  approximated  by  bounding  polygons  shown  in  Figure  4.  This  technique  using 
polygon  templates  was  initially  used  by  Fused  Reality  and  appeared  in  the  original  patent  application. 
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Figure  2:  Snapshot  of  Magenta  Background. 
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Figure  3:  Red  vs.  Green  and  Red  vs.  Blue  Scatterplots  of 
Camera  Response  to  the  Magenta  in  Figure  2. 


Figure  4:  Polygons  Bounding  Red  vs.  Green  and  Red  vs.  Blue  Scatterplots. 


A  new  technique  that  uses  a  less  complex  color  decomposition  has  since  been  identified  and  is  now  used  by 
Fused  Reality  to  define  a  surface’s  color.  Figure  5  shows  the  pixel  scatterplots  for  these  color  coordinates 
corresponding  to  the  image  in  Figure  2.  Note  that  these  coordinates  produce  scatterplots  that  can  be  defined 
via  bands  (instead  of  complex  and  relatively  imprecise  polygons)  based  on  their  probability  densities,  shown 
below  the  scatterplots.  Thus  it  is  possible  to  statistically  define  the  color  characteristics  of  a  relatively  an 
image  simply  through  lower  and  upper  boundaries  -  a  much  simpler  process  than  the  RGB  mapping  (which 
requires  linear  interpolation)  shown  in  Figure  4.  A  modification  to  the  Fused  Reality  patent  is  incorporating 
this  new  technique. 
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Figure  5:  ScatterPlots  and  their  Probability  Densities  Using  the  New  Color  Coordinate  System. 

The  robustness  of  this  technique  is  demonstrated  in  Figure  6,  where  a  magenta  surface  (shown  top)  mounted 
serves  as  a  virtual  display.  The  bottom  photos  in  Figure  6  show  two  very  different  lighting  environments 
(note  the  desk  reflection  brightness),  but  the  magenta  is  correctly  identified  despite  the  variation  in  lighting. 


Figure  6:  Raw  Video  Showing  Magenta  Board  (top),  Fused  Reality  under 
Low  Lighting  (lower  left),  and  Bright  Lighting  (lower  right)  Conditions. 
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The  advantages  of  Fused  Reality  and  its  current  recognition  scheme  thus  include:  1)  Backdrop  can  be  made 
from  cheap  cotton  sheet;  2)  Chromakey  is  independent  of  viewing  angle:  3)  It  can  use  any  lighting 
(incandescent,  fluorescent)  and  brightness  that  makes  the  backdrop  visually  distinguishable  from  its 
surroundings;  4)  More  than  one  target  color  can  be  used;  5)  Lighting  level  does  not  have  to  be  low  to  simulate 
low  light  visual  environments  or  Night  Vision,  as  each  pixel  can  be  operated  on  to  change  light  level  or  color 
displayed  to  user.  Fused  Reality  segments  space  into  the  near  tangible  environment  and  the  distant  virtual 
environment,  so  that  high  perceptual  fidelity  is  maintained  in  both  domains  while  minimizing  computational 
expense:  a)  The  user  naturally  encounters  the  high  detail  of  the  physical  world  (vision,  touch);  b)  Excellent 
perception  of  the  distant  virtual  world  generally  requires  low  to  medium  level  of  detail. 


VISUAL  SYSTEM 

A  preliminary  visual  system  is  shown  in  Figure  7,  where  a  Cohu  camera  has  been  mounted  onto  an  eMagin 
HMD.  A  flexible  cable  connects  the  sensor  (mounted  center  and  front)  with  the  control  box  (mounted  on  the 
left  hand  temple).  A  12mm  Computar  lens  is  shown  mounted  on  the  sensor.  In  order  to  eliminate  parallax 
during  manual  tasks  (such  as  gunnery)  a  future  design  will  mount  the  camera  level  with  the  eyes  in  front  of 
the  HMD. 


Figure  7:  Cohu  Camera  Mounted  on  the  eMagin  HMD. 


In  Figure  8  the  Cohu  camera  is  shown  mounted  on  an  Air  Force  helmet  that  is  currently  being  used  by  the  Air 
National  Guard  with  STI’s  Parachute  Simulator,  ParaSim®.  The  eMagin  HMD  attaches  onto  the  helmet’s 
NVG  clip  via  a  swivel  mount. 


Figure  8:  Cohu  Camera  Mounted  on  a  Gentex  Helmet  with  an  NVG  Clip. 
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In  Figure  9a  a  magenta-colored  panel  is  shown  hung  on  a  wall,  and  Figure  9b  shows  the  magenta  pixels 
replaced  with  a  real-time  virtual  marine  scene.  Figure  9c  demonstrates  Fused  Reality’s  capability  to:  1)  allow 
physical  objects  that  occlude  the  magenta  background  to  be  viewed,  and  2)  precisely  discriminate  between  the 
magenta  target  color  and  flesh-tone  colors. 


Figure  9:  a)  Wall-Mounted  Magenta  Panel;  b)  Magenta  Surface  Becomes  a  Virtual  Display  Showing  a 
Marine  Scene  with  Navy  Ship;  c)  Hand  Placed  between  the  Viewer’s  Camera  and  Magenta  Backdrop. 

A  recent  helicopter  gunnery  demonstration  to  the  Navy  was  conducted  for  the  PA  VET  SBIR.  Figure  10 
shows  the  raw  and  fused  video  of  the  cabin  environment  (constructed  by  Binghamton  Simulator  Company) 
wth  a  virtual  background  scene. 

Lumakey 

Lumakey  is  a  patented  (pending)  process  that  employs  the  inversion  of  Fused  Reality  logic  by  rendering  all 
pixels  transparent  unless  a  brightness  criterion  is  met.  The  user  would  be  placed  in  a  darkened  room  and  a 
head-mounted  light  would  illuminate  physical  objects  within  reach.  This  concept  is  demonstrated  in  Figure 
11.  Lumakey  would  be  best-suited  for  applications  where:  1)  The  user  remains  fixed  in  position;  2)  User 
physically  interacts  with  objects  that  are  within  arm’s  reach;  3)  The  surrounding  near-space  environment  is 
sparse;  and  4)  High  fidelity,  real  time  viewing  of  hand  and  equipment  required.  Figure  12  shows  such  an 
application,  where  a  parachute  trainee  is  suspended  in  STI’s  Parasim. 
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Raw  Video  Fused  Video 


Figure  10:  PAVET  Cabin  Photos  of  Raw  and  Fused  Video. 


Figure  11:  Fused  Reality  Using  Lumakey. 
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Figure12:  Trainee  Suspended  from  Parasim  Scaffolding. 

Deployment  of  Physical  Objects  in  the  Virtual  World 

Blue  screen  production  places  and  moves  virtual  objects  within  a  physical  setting.  However,  Fused  Reality 
offers  a  unique  capability:  transporting  physical  objects  within  a  virtual  environment.  This  is  accomplished 
by  taking  real-time  bitmaps  of  actual  objects  and  pasting  them  onto  virtual,  simple3D  polyhedrons  that  can  be 
moved  in  the  virtual  environment.  Thus  a  user  can  deploy  hand-held  physical  objects  into  virtual  space  and 
retrieve  them,  with  the  objects  interacting  in  sync  with  other  real  objects/participants.  This  obviates  need  for 
computationally  expensive  simulation  of  complex  3D  models  (and  the  creation  of  those  models).  Figure  13 
demonstrates  this  concept,  where  a  figurine  is  suspended  from  a  wire  that  can  be  rotated  to  yaw  the  figurine 
about  its  vertical  axis  -  note  that  the  physical  position  of  the  object  is  fixed.  Figures  13a  and  13b  show  the 
scaffolding  and  magenta  background.  In  Figures  13c-f  a  user  controls  the  virtual  position  of  the  figurine 
using  a  joystick  while  the  figurine  physically  pirouettes  (the  wire  is  rotated).  The  real-time  bitmap  sent  by  the 
camera  is  pasted  onto  a  billboard  which  is  translated  into  the  3D  virtual  environment  via  the  joystick 
commands. 

This  technique  could  be  employed  in  helicopter  rescue  litter  training,  where  the  physical  litter  can  be  hooked 
up  to  a  hoist,  pushed  out  of  the  cabin,  and  lowered  on  a  cable  for  a  few  feet.  After  the  litter  ceases  to 
physically  lower  (because  of  space  limitations)  the  bit  map  of  the  suspended  litter  would  be  fused  to  a 
horizontal  billboard,  which  would  then  continue  its  descent  in  the  virtual  following  the  control  commands  of 
the  aircrewman.  It  must  be  remembered  that  the  user  would  be  viewing  the  actual  image  of  the  litter , 
swaying  and  rotating  as  it  is  suspended  from  the  physical  cable.  When  the  virtual  survivor  approaches  the 
litter,  an  actual  person  can  step  into  it  (which  is  near  the  floor)  so  that  the  trainee  feels  the  full  weight  on  the 
hoist  cable,  and  the  person’s  bitmap  would  be  fused  to  the  litter’s  billboard.  During  retrieval  of  the 
litter/survivor,  the  virtually  transported  bitmap  would  ascend  until  it  reached  the  distance  of  the  actual 
litter/survivor,  at  which  point  the  bit  map  would  be  detached  from  the  virtual  billboard  and  the  aircrewman 
would  observe  the  physical  litter/survivor  for  the  remaining  few  feet  and  be  able  to  pull  them  both  into  the 
cabin.  Figure  14  shows  a  photo  of  the  litter  used  for  Navy  rescue. 
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Figure  13:  Frames  Showing  Physical  Object  (witch  figurine)  Translated  in  Virtual  Space. 


27-10 


RTO-MP-HFM-136 


Helicopter  Aircrew  Training  Using  Fused  Reality 


Figure  14:  Navy  Rescue  Litter. 


ALTERNATE  TECHNOLOGIES  TO  FUSED  REALITY 

Z-key  is  an  image  keying  method  that  uses  pixel-by-pixel  depth  information  of  real  and  virtual  objects  to  form 
of  a  depth  map,  allowing  separation  of  mixed  reality  objects  based  on  the  detailed  shape  and  position  of 
surfaces.  Figure  1  is  a  schema  of  a  z-key  method  that  uses  stereographic  information  as  a  range-finding 
mechanism.  A  close-up  of  the  merged  images  shows  the  border  of  the  real  objects  to  be  very  ragged,  due  to 
the  noisy  stereographic  feedback  at  the  edges.  The  frame  rate  using  this  technique  is  roughly  15  frames/sec, 
compared  to  30  frames/sec  using  Fused  Reality  (soon  to  be  60  Hz).  Additionally,  a  featureless  background 
such  as  a  smooth  wall  would  not  provide  depth  information  to  this  system. 


Range-finding  user  laser  is  precise,  but  it  gives  very  low  frame  rates. 


outpul  image  IQ(ij]I 


Figure  15:  Z-Keying  Process  Using  Stereographic  Feedback  [2]. 
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Figure  16:  Merging  of  Physical  and  Synthetic  Near-Space  Objects  Using  Z-Keying  [2]. 

Chromakey  using  a  retroreflective  background  and  a  LED-ringed  camera  is  another  technique  that  has 
recently  been  useful  in  filming.  However,  when  a  user’s  hands  disrupt  the  light  from  a  helmet-mounted  light 
source,  the  shadows  cast  onto  the  distant  background  are  diffuse  and  poorly  defined.  This  in  turn  degrades  the 
chromakey  process  near  the  borders  of  the  subjects.  In  the  film  industry  this  is  not  an  issue  as  the  camera  is 
positioned  much  further  away  from  the  subjects  than  the  subjects’  distances  from  the  background.  Moreover, 
retroreflection  yields  poor  results  when  the  viewing  angle  is  very  oblique  to  the  background  plane. 


Figure  17:  Retroreflective  Background  and  LED-Ringed  Camera. 

Head  Mounted  Projection  Displays  (HMPD’s)  [3]  also  make  used  of  a  head-mounted  light  source  and 
retroreflective  screens,  and  have  constraints  similar  to  those  given  above. 

FUTURE  FEATURES  USING  FUSED  REALITY 

Target  areas  (portals)  could  be  designated  by  infrared  (IR)  and  ultraviolet  (UV)  reflection,  giving  rise  to 
virtual  reality  portal  generation  on-command  (i.e.  directing  an  IR  or  UV  source  toward  a  reflective  surface). 
Dual  reality  portals  could  be  made  by  coating  glass  with  transparent  material  that  reflects  IR  or  UV  light,  so 
that  the  trainee  sees  a  virtual  environment  while  the  naked-eye  observer  can  view  the  actual  environment  that 
exists  beyond  the  glass.  This  would  allow  training  in  actual  vehicles  such  as  cars  (driven  in  vacant  parking 
lots)  and  aircraft  while  a  safety  observer  looks  for  potential  conflicts  or  other  hazards.  Thus  a  trainee  would 
experience  the  actual  forces  that  he/she  is  effecting  while  operating  in  a  virtual  world  such  as  the  one  shown  in 
Figure  18. 
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Figure  18:  View  from  STISIM. 


[1]  Dean,  Frank,  P.  Garrity  and  C.  Stapleton,  Mixed  Reality:  A  Tool  for  Integrating  Live ,  Virtual  & 
Constructive  Domains  to  Support  Training  Transformation ,  Interservice/Industry  Training,  Simulation 
and  Education  Conference  (I/ITSEC)  2004. 

[2]  http://www.cs.cmu.edu/afs/cs/project/stereo-machine/www/z-key.html 

[3]  Nguyen,  Long,  A.  Mead,  C.  Fidopiastis,  and  J.  Rolland,  Enhancing  Virtual  Environment  Interaction 
Using  a  Head  Mounted  Projection  Display  ( HMPD ),  Interservice/Industry  Training,  Simulation  and 
Education  Conference  (I/ITSEC)  2004. 
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Initial  Motivation  for  FR 


Helicopter  cabin  crew  should  interact  seamlessly  with  the 
flight  crew  and  dynamic  external  environment 

Due  to  physical  constraints  and  fidelity  limitations,  current 
simulation  designs  fail  to  accommodate  the  wide  range  of 
training  required  for  multi-role  helicopter  platforms 

-  Search  and  rescue 

-  Vertical  replenishment  (cargo  hook  operations) 

-  Inflight  refueling 

-  Flight  operation  during  whiteout  and  brownout 


15  Jun  2006 


HFM  Panel  Workshop 


2 


Candidate  Simulation  Approaches 


Virtual  Reality 

-  Tactile  interaction  with  objects  in  a  physical  environment  requires 
distance  sensing  between  the  human  touchpoints  (hand,  elbow, 
etc.)  and  all  objects  that  can  be  touched 

-  Intractable  problem  for  complex  manual  tasks  such  as  component 
disassembly 

-  Even  for  relatively  simple  manual  tasks  such  as  flight  control 
manipulation,  user  cannot  see  actual  positioning  of  limbs  and 
hands  (important  elements  for  manual  control)  unless  the  body  is 
peppered  with  sensors 

-  User  cannot  perceive  other  players  realistically 

>  Facial  expressions 

>  Body  posture 

>  Clothing 
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Candidate  Simulation  Approaches 


CAVE  (Computer  Animated  Virtual  Environment) 

-  Hemispherical  dome  display  placed  outside  the  cabin  so  that 
trainee  will  see  a  projected  environment  (with  the  naked  eye) 
beyond  the  cabin  airframe 

>  Limited  volume 

>  Expensive 

>  Requires  a  display  wherever  you  want  an  image 

>  Cannot  process  real  objects  (such  as  deploying  them  into  the  virtual 
environment) 

>  Power  intensive  (hot,  component  failure) 
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Alternative  Approach: 

Fused  Reality 

Real  time  interactive  fusion  of  physical  and  virtual  environments 
Employs  three  proven  technologies 

1.  Live  video  capture 

2.  Real-time  video  editing 

3.  Virtual  environment  simulation 

Segments  space  into: 

1 .  The  near  tangible  environment 

2.  Distant  virtual  environment 

High  perceptual  fidelity  is  maintained  in  both  domains  while 
minimizing  computational  expense: 

1 .  User  naturally  encounters  the  high  detail  of  the  physical  world  (vision,  touch) 

2.  Excellent  perception  of  the  distant  virtual  world  generally  requires  low  to 
medium  level  of  detail 
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Fused  Reality  Representation 
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Helmet  Mounted  Display  &  Camera 


Cohu  Camera 

-  30  frames/sec 

-  720  x  480  resolution 

-  $800 


eMagin  Z800  HMD 

-  40  deg  diag  FOV 

-  800  x  600  res 

-  $800 
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FR  Key  Features 


•  Portal  areas  fixed  (fused)  to  the  physical  visual  environment 

-  Not  based  on  head  orientation/position,  which  is  subject  to  tracking: 

1.  Noise 

2.  Drift 

3.  Lag 

4.  Masking 

•  Incorporates  an  STI-patented  process  for  robustly  and  quickly 
sensing  target  color  under  a  wide  range  of  lighting  and  surface 
irregularities 

-  Backdrop  is  made  from  cheap  cotton  sheet 

-  Independent  of  viewing  angle 

-  Any  lighting  (incandescent,  fluorescent)  and  brightness  that  makes  the 
backdrop  visually  distinguishable  from  its  surroundings 
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ADAPTIVE  COLOR  RECOGNITION 
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FR  Key  Features,  cont’d 


Operators  can  view  real  objects  (such  as  hands,  equipment) 
which  are  overlaid  on  the  portal  areas 

Uses  magenta  as  target  color 

-  Color  rarely  encountered  in  simulation  environments  (i.e.  cockpit) 

-  Clearly  identifies  magenta  from  flesh-tones 

Can  use  more  than  one  target  color  in  real  time 

-  Avoid  proximity  to  objects  having  similar  colors 

-  Target  color  selection  can  be  based  on  head  orientation/position 

Lighting  level  does  not  have  to  be  low  to  simulate  low  light  visual 
environments  or  Night  Vision 

-  Can  operate  on  each  pixel  to  change  light  level  or  color  displayed  to  user 
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Fused  Reality 
Gunnery  Task 
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FR  Key  Features, 


cont’d 


•  User  can  deploy  hand-held  physical  objects 
into  virtual  space  and  retrieve  them 


-  Real  time  bitmaps  of  actual  objects  used 


-  Deployed  object  interacts  in  sync  with  other  real 
objects/participants 


> 


Rescue  litter  operations 


-  Removes  need  for  computationally  expensive 
simulation  of  complex  3D  models  (and  the  creation 
of  those  models) 


•  For  geometrically  &  dyamically  simple 
objects: 

-  Virtual  model  can  replace  the  physical  object  when 
out  of  user’s  reach 


-  Physical  object  replace  the  virtual  model  when 
within  user’s  reach 
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Lumakey 

•  Coined  the  word  earlier  this  year  (appearing  in  patent) 

•  Inversion  of  current  FR  logic 

-  Make  all  pixels  transparent  unless  criterion  met 

•  Criterion 

-  Brightness  level  -  that’s  it! 

•  Ideal  for  applications  where: 

-  User  remains  fixed  in  position 

-  User  physically  interacts  with  objects  that  are  within  arm’s  reach 

-  Surrounding  near-space  environment  is  sparse 

-  High  fidelity,  real  time  viewing  of  hand  and  equipment  required 
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Lumakey,  cont’d 


•  Potential  applications: 

-  Parachute  simulation 

-  Crane  operation  simulation 

-  Teleoperation 
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FR  Limitations 


User’s  field-of-view  (FOV)  limited  by  HMD 

Current  video  sensor  resolution  does  not  provide  20/20  vision 

-  If  a  monitor  is  being  observed,  losses  are  minimized  by  routing  the  monitor’ 
video  signal  directly  to  the  image  generator  and  placing  a  magenta  panel  in 
the  monitor’s  location 

Does  not  accommodate  for  virtual  objects  operating  in  the  near 
(physical)  space 

-  Haptic  feedback  required  for  realistic  interaction  with  virtual  objects 
(currently  not  feasible  except  for  very  special  task  scenarios) 

FR  focuses  on  physical  interaction  with  the  near-space  that 
effects: 

1 .  Relative  motion  and  position  with  far-space 

2.  Events  in  far-space 
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Alternate  Technologies 


•  Blue-screening  using  retro-reflective  screens 


-  Highly  reflective  (similar  to  the  function  of  materials  in  street  signs) 


-  Requires  a  blue  light  source  originating  from  user’s  viewpoint  (such  as 
camera-ringed  LED’s) 


-  Image  will  be  interrupted  when  objects  such  user’s  hand  comes  between 
light  source  and  screen 


>  Unless  occluding  object  is  very  close  to  the  screen,  object’s  shadow 
becomes  ill-defined  and  inconsistent  with  the  actual  object 

•  Head-projected  images 


-  Wide  FOV 


-  No  image  processing  required 

-  Uses  retro-reflective  screen  material,  subject  to  same  occlusion  aberrations 
(object  must  be  very  close  to  the  retro-reflective  screen  when  occlusion 
occurs) 
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Z-Keying 


Use  stereo  image  information  for  determining  depth  of  real 
objects 


-  Computationally  intensive,  low  frame  rate 

-  Not  suitable  when  physical  environment  contains  untextured  surfaces 

Laser  ranging 

-  Resolution/data  rate  tradeoff  not  compatible  with  real  time  operation 
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Future 
FR  Features 


•  Target  areas  designated  by  infrared  (IR)  and  ultraviolet  (UV) 
reflection 

-  VR  portal  generation  on-command  (IR-directed) 

•  Dual  reality  portals 

-  Safety  observer  views  real  out-the-window  world  through  naked-eye 
transparent  windows 

-  VR  portals  created  by  transparent,  IR/UV  reflective  window  coating 

-  User  experiences  actual  forces  that  result  from  control  inputs  and  external 
disturbances  (driving,  flight,  shipboard  applications) 

•  Stereo  vision  for  precision  eye-hand  coordination  tasks 
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Technology  Insertion  Areas  for  FR 

•  Medevac 

•  Dismounted/Mounted  Combat 

•  Fire  fighting 

•  Surgery  simulation  and  medical  therapy 

•  Driving 

•  Sports 

•  Entertainment  Industry 

•  Embedded  simulation 
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